{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## [Chapter 3] Controlling Relevance\n",
    "\n",
    "In the [last notebook](1.vectors-and-text-similarity.ipynb), we introduced how queries and documents can be represented as vectors, how cosine similarity can be used as a relevance function to compare queries and documents, and how tf-idf ranking can be used to create a feature weight that balances both strength of occurrence (TF) and significance of a term (IDF) for each term in a term-based vector.\n",
    "\n",
    "In this notebook, we will show how a full relevance function can be specified and controlled in a search engine (Apache Solr). Let's start with showing off the default similarity calculation leveraged by all Lucene-based search engines: BM25\n",
    "\n",
    "### BM25 (Best Match Okapi 25)\n",
    "The BM25 algorithm is the default similarity function in Apache Lucene, Apache Solr, Elasticsearch, Lucidworks Fusion, and other Lucene-based search engines. The algorithm leverages TF and IDF, but also incorporates many additional configurable options. The full BM25 calculation is found in Figure 3.7 in the book.\n",
    "\n",
    "Instead of reimplementing the full BM25 algorithm in Python, let's go ahead and switch over to using our search engine (Apache Solr) and see how it performs the calculation."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Creating our Collection"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Listing 3.9"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Wiping \"cat_in_the_hat\" collection\n",
      "Creating \"cat_in_the_hat\" collection\n",
      "Status: Success\n"
     ]
    }
   ],
   "source": [
    "from aips import *\n",
    "\n",
    "engine = get_engine()\n",
    "collection = engine.create_collection(\"cat_in_the_hat\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Add some documents"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Listing 3.10"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Adding Documents to 'cat_in_the_hat' collection\n",
      "Status: Success\n"
     ]
    }
   ],
   "source": [
    "docs = [\n",
    "    {\n",
    "        \"id\": \"doc1\",\n",
    "        \"title\": \"Worst\",\n",
    "        \"description\": \"The interesting thing is that the person in the wrong made the right decision in the end.\"\n",
    "    },\n",
    "    {\n",
    "        \"id\": \"doc2\",\n",
    "        \"title\": \"Best\",\n",
    "        \"description\": \"My favorite book is the cat in the hat, which is about a crazy cat who breaks into a house and creates a crazy afternoon for two kids.\"\n",
    "        \n",
    "    },\n",
    "    {\n",
    "        \"id\": \"doc3\",\n",
    "        \"title\": \"Okay\",\n",
    "        \"description\": \"My neighbors let the stray cat stay in their garage, which resulted in my favorite hat that I let them borrow being ruined.\"        \n",
    "    }\n",
    "]\n",
    "response = collection.add_documents(docs)\n",
    "engine.print_status(response)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Inspecting BM25 Score\n",
    "Now, let's execute a query for `the cat in the hat`, and see how each document scores using the BM25 similarity calculation:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Listing 3.11"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<strong>Query</strong>: <i>the cat in the hat</i><br/><br/><strong>Results:</strong>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "[{'id': 'doc2',<br/>'title': 'Best',<br/>'description': 'My favorite book is the cat in the hat, which is about a crazy cat who breaks into a house and creates a crazy afternoon for two kids.',<br/>'score': 0.68231964,<br/>'[explain]': '<ul><li>0.68231964 = sum of:<br /><ul><li>0.27551937 = weight(description:cat in 1) [SchemaSimilarity], result of:<br /><ul><li>0.27551937 = score(freq=2.0), computed as boost * idf * tf from:<br /><ul><li>0.47000363 = idf, computed as log(1 + (N - n + 0.5) / (n + 0.5)) from:<br /><ul><li>2 = n, number of documents containing term<br /></li></ul><ul><li>3 = N, total number of documents with field<br /></li></ul></li></ul><ul><li>0.5862069 = tf, computed as freq / (freq + k1 * (1 - b + b * dl / avgdl)) from:<br /><ul><li>2.0 = freq, occurrences of term within document<br /></li></ul><ul><li>1.2 = k1, term saturation parameter<br /></li></ul><ul><li>0.75 = b, length normalization parameter<br /></li></ul><ul><li>28.0 = dl, length of field<br /></li></ul><ul><li>22.666666 = avgdl, average length of field<br /></li></ul></li></ul></li></ul></li></ul><ul><li>0.055366673 = weight(description:in in 1) [SchemaSimilarity], result of:<br /><ul><li>0.055366673 = score(freq=1.0), computed as boost * idf * tf from:<br /><ul><li>0.13353139 = idf, computed as log(1 + (N - n + 0.5) / (n + 0.5)) from:<br /><ul><li>3 = n, number of documents containing term<br /></li></ul><ul><li>3 = N, total number of documents with field<br /></li></ul></li></ul><ul><li>0.4146341 = tf, computed as freq / (freq + k1 * (1 - b + b * dl / avgdl)) from:<br /><ul><li>1.0 = freq, occurrences of term within document<br /></li></ul><ul><li>1.2 = k1, term saturation parameter<br /></li></ul><ul><li>0.75 = b, length normalization parameter<br /></li></ul><ul><li>28.0 = dl, length of field<br /></li></ul><ul><li>22.666666 = avgdl, average length of field<br /></li></ul></li></ul></li></ul></li></ul><ul><li>0.19487953 = weight(description:hat in 1) [SchemaSimilarity], result of:<br /><ul><li>0.19487953 = score(freq=1.0), computed as boost * idf * tf from:<br /><ul><li>0.47000363 = idf, computed as log(1 + (N - n + 0.5) / (n + 0.5)) from:<br /><ul><li>2 = n, number of documents containing term<br /></li></ul><ul><li>3 = N, total number of documents with field<br /></li></ul></li></ul><ul><li>0.4146341 = tf, computed as freq / (freq + k1 * (1 - b + b * dl / avgdl)) from:<br /><ul><li>1.0 = freq, occurrences of term within document<br /></li></ul><ul><li>1.2 = k1, term saturation parameter<br /></li></ul><ul><li>0.75 = b, length normalization parameter<br /></li></ul><ul><li>28.0 = dl, length of field<br /></li></ul><ul><li>22.666666 = avgdl, average length of field<br /></li></ul></li></ul></li></ul></li></ul><ul><li>0.15655404 = weight(description:the in 1) [SchemaSimilarity], result of:<br /><ul><li>0.15655404 = score(freq=2.0), computed as boost * idf * tf from:<br /><ul><li>2.0 = boost<br /></li></ul><ul><li>0.13353139 = idf, computed as log(1 + (N - n + 0.5) / (n + 0.5)) from:<br /><ul><li>3 = n, number of documents containing term<br /></li></ul><ul><li>3 = N, total number of documents with field<br /></li></ul></li></ul><ul><li>0.5862069 = tf, computed as freq / (freq + k1 * (1 - b + b * dl / avgdl)) from:<br /><ul><li>2.0 = freq, occurrences of term within document<br /></li></ul><ul><li>1.2 = k1, term saturation parameter<br /></li></ul><ul><li>0.75 = b, length normalization parameter<br /></li></ul><ul><li>28.0 = dl, length of field<br /></li></ul><ul><li>22.666666 = avgdl, average length of field<br /></li></ul></li></ul></li></ul></li></ul></li></ul>'}, {'id': 'doc3',<br/>'title': 'Okay',<br/>'description': 'My neighbors let the stray cat stay in their garage, which resulted in my favorite hat that I let them borrow being ruined.',<br/>'score': 0.62850046,<br/>'[explain]': '<ul><li>0.62850046 = sum of:<br /><ul><li>0.21236044 = weight(description:cat in 2) [SchemaSimilarity], result of:<br /><ul><li>0.21236044 = score(freq=1.0), computed as boost * idf * tf from:<br /><ul><li>0.47000363 = idf, computed as log(1 + (N - n + 0.5) / (n + 0.5)) from:<br /><ul><li>2 = n, number of documents containing term<br /></li></ul><ul><li>3 = N, total number of documents with field<br /></li></ul></li></ul><ul><li>0.45182723 = tf, computed as freq / (freq + k1 * (1 - b + b * dl / avgdl)) from:<br /><ul><li>1.0 = freq, occurrences of term within document<br /></li></ul><ul><li>1.2 = k1, term saturation parameter<br /></li></ul><ul><li>0.75 = b, length normalization parameter<br /></li></ul><ul><li>23.0 = dl, length of field<br /></li></ul><ul><li>22.666666 = avgdl, average length of field<br /></li></ul></li></ul></li></ul></li></ul><ul><li>0.08311336 = weight(description:in in 2) [SchemaSimilarity], result of:<br /><ul><li>0.08311336 = score(freq=2.0), computed as boost * idf * tf from:<br /><ul><li>0.13353139 = idf, computed as log(1 + (N - n + 0.5) / (n + 0.5)) from:<br /><ul><li>3 = n, number of documents containing term<br /></li></ul><ul><li>3 = N, total number of documents with field<br /></li></ul></li></ul><ul><li>0.6224256 = tf, computed as freq / (freq + k1 * (1 - b + b * dl / avgdl)) from:<br /><ul><li>2.0 = freq, occurrences of term within document<br /></li></ul><ul><li>1.2 = k1, term saturation parameter<br /></li></ul><ul><li>0.75 = b, length normalization parameter<br /></li></ul><ul><li>23.0 = dl, length of field<br /></li></ul><ul><li>22.666666 = avgdl, average length of field<br /></li></ul></li></ul></li></ul></li></ul><ul><li>0.21236044 = weight(description:hat in 2) [SchemaSimilarity], result of:<br /><ul><li>0.21236044 = score(freq=1.0), computed as boost * idf * tf from:<br /><ul><li>0.47000363 = idf, computed as log(1 + (N - n + 0.5) / (n + 0.5)) from:<br /><ul><li>2 = n, number of documents containing term<br /></li></ul><ul><li>3 = N, total number of documents with field<br /></li></ul></li></ul><ul><li>0.45182723 = tf, computed as freq / (freq + k1 * (1 - b + b * dl / avgdl)) from:<br /><ul><li>1.0 = freq, occurrences of term within document<br /></li></ul><ul><li>1.2 = k1, term saturation parameter<br /></li></ul><ul><li>0.75 = b, length normalization parameter<br /></li></ul><ul><li>23.0 = dl, length of field<br /></li></ul><ul><li>22.666666 = avgdl, average length of field<br /></li></ul></li></ul></li></ul></li></ul><ul><li>0.120666236 = weight(description:the in 2) [SchemaSimilarity], result of:<br /><ul><li>0.120666236 = score(freq=1.0), computed as boost * idf * tf from:<br /><ul><li>2.0 = boost<br /></li></ul><ul><li>0.13353139 = idf, computed as log(1 + (N - n + 0.5) / (n + 0.5)) from:<br /><ul><li>3 = n, number of documents containing term<br /></li></ul><ul><li>3 = N, total number of documents with field<br /></li></ul></li></ul><ul><li>0.45182723 = tf, computed as freq / (freq + k1 * (1 - b + b * dl / avgdl)) from:<br /><ul><li>1.0 = freq, occurrences of term within document<br /></li></ul><ul><li>1.2 = k1, term saturation parameter<br /></li></ul><ul><li>0.75 = b, length normalization parameter<br /></li></ul><ul><li>23.0 = dl, length of field<br /></li></ul><ul><li>22.666666 = avgdl, average length of field<br /></li></ul></li></ul></li></ul></li></ul></li></ul>'}, {'id': 'doc1',<br/>'title': 'Worst',<br/>'description': 'The interesting thing is that the person in the wrong made the right decision in the end.',<br/>'score': 0.3132525,<br/>'[explain]': '<ul><li>0.3132525 = sum of:<br /><ul><li>0.089769006 = weight(description:in in 0) [SchemaSimilarity], result of:<br /><ul><li>0.089769006 = score(freq=2.0), computed as boost * idf * tf from:<br /><ul><li>0.13353139 = idf, computed as log(1 + (N - n + 0.5) / (n + 0.5)) from:<br /><ul><li>3 = n, number of documents containing term<br /></li></ul><ul><li>3 = N, total number of documents with field<br /></li></ul></li></ul><ul><li>0.67226887 = tf, computed as freq / (freq + k1 * (1 - b + b * dl / avgdl)) from:<br /><ul><li>2.0 = freq, occurrences of term within document<br /></li></ul><ul><li>1.2 = k1, term saturation parameter<br /></li></ul><ul><li>0.75 = b, length normalization parameter<br /></li></ul><ul><li>17.0 = dl, length of field<br /></li></ul><ul><li>22.666666 = avgdl, average length of field<br /></li></ul></li></ul></li></ul></li></ul><ul><li>0.2234835 = weight(description:the in 0) [SchemaSimilarity], result of:<br /><ul><li>0.2234835 = score(freq=5.0), computed as boost * idf * tf from:<br /><ul><li>2.0 = boost<br /></li></ul><ul><li>0.13353139 = idf, computed as log(1 + (N - n + 0.5) / (n + 0.5)) from:<br /><ul><li>3 = n, number of documents containing term<br /></li></ul><ul><li>3 = N, total number of documents with field<br /></li></ul></li></ul><ul><li>0.83682007 = tf, computed as freq / (freq + k1 * (1 - b + b * dl / avgdl)) from:<br /><ul><li>5.0 = freq, occurrences of term within document<br /></li></ul><ul><li>1.2 = k1, term saturation parameter<br /></li></ul><ul><li>0.75 = b, length normalization parameter<br /></li></ul><ul><li>17.0 = dl, length of field<br /></li></ul><ul><li>22.666666 = avgdl, average length of field<br /></li></ul></li></ul></li></ul></li></ul></li></ul>'}]"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "query = \"the cat in the hat\"\n",
    "request = {\"query\": query,\n",
    "           \"query_fields\": [\"description\"],\n",
    "           \"return_fields\": [\"id\", \"title\", \"description\", \"score\"],\n",
    "           \"explain\": True}\n",
    "\n",
    "response = collection.search(**request)\n",
    "\n",
    "display_search(query, response[\"docs\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "While the BM25 calculation is much more complex than the TF-IDF feature weight calculations we saw in the [last notebook](ch3-vectors-and-text-similarity.ipynb), it is fundamentally still derived from TF-IDF at it's core. As such, you'll notice that the search results actually return in the exact same order as our TF-IDF calculations from the last notebook:\n",
    "\n",
    "<pre>\n",
    "doc2: 0.6823196\n",
    "doc3: 0.6285005\n",
    "doc1: 0.3132525\n",
    "</pre>\n",
    "\n",
    "Our query for *the cat in the hat* can still very much be thought of as a vector of the BM25 scores for each of the terms: [\"the\", \"cat\", \"in\", \"the\", \"hat\"].\n",
    "\n",
    "What may not be obvious, however, is that the feature weights for each of these features are actually just overridable functions. For example, this query could alternatively be expressed as the vector:\n",
    "\n",
    "<pre>\n",
    "[ query(\"the\"), query(\"cat\"), query(\"in\"), query(\"the\"), query(\"hat\") ]\n",
    "</pre>\n",
    "\n",
    "In Solr syntax, this would be:\n",
    "\n",
    "<pre>\n",
    "{\"query\": '{!func}query(\"the\") {!func}query(\"cat\") {!func}query(\"in\") {!func}query(\"the\") {!func}query(\"hat\")'}\n",
    "</pre>\n",
    "\n",
    "For that query, we get the following:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Listing 3.12"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<strong>Query</strong>: <i>{!func}query(\"the\") {!func}query(\"cat\") {!func}query(\"in\") {!func}query(\"the\") {!func}query(\"hat\")</i><br/><br/><strong>Results:</strong>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "[{'id': 'doc2',<br/>'title': 'Best',<br/>'score': 0.68231964}, {'id': 'doc3',<br/>'title': 'Okay',<br/>'score': 0.62850046}, {'id': 'doc1',<br/>'title': 'Worst',<br/>'score': 0.3132525}]"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "query = '{!func}query(\"the\") {!func}query(\"cat\") {!func}query(\"in\") {!func}query(\"the\") {!func}query(\"hat\")'\n",
    "request = {\"query\": query,\n",
    "           \"return_fields\": [\"id\", \"title\", \"score\"],\n",
    "           \"query_fields\": \"description\"}\n",
    "\n",
    "response = collection.search(**request)\n",
    "\n",
    "display_search(query, response[\"docs\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The scores are exactly the same! Not only that, but once we realize that every term in a query is nothing more than a configurable scoring function, it opens up tremendous possibilities for manipulating that scoring function.\n",
    "\n",
    "### Functions, Functions, Everywhere!\n",
    "Now that we've seen that the relevance score for each term in our queries is simply a function operating on that term to generate a feature weight, the next logical question is \"what OTHER kinds of functions can I use in my queries?\".\n",
    "\n",
    "We've already encountered the *query* function, which is effectively the default calculation that executes whenever no explicit function is specified, and which uses the BM25 similarity algorithm by default.\n",
    "\n",
    "But what if we want to consider some other features in our scoring calculation, perhaps some that are not text-based?\n",
    "\n",
    "Here is a partial list of common relevance techniques:\n",
    "- Geospatial Boosting (documents near the user running the query should rank higher)\n",
    "- Date Boosting (Newer documents should get a higher relevancy boost)\n",
    "- Popularity Boosting (Documents which are more popular should get a higher relevancy boost)\n",
    "- Field Boosting (Terms matching in certain fields should get a higher weight than in other fields)\n",
    "- Category Boosting (Documents in categories related to query terms should get a higher relevancy boost)\n",
    "- Phrase Boosting (documents matching multi-term phrases in the query should rank higher than those only matching the words separately)\n",
    "- ...\n",
    "\n",
    "Many of these techniques are built into specific query parsers in Solr, either through query syntax or through query parser options. For example, field boosting can be accomplished through the `qf` parameter on the `edismax` query parser:\n",
    "\n",
    "  <pre>\n",
    "  {\"query\": \"the cat in the hat\",\n",
    "   <b>\"fields\": [\"title^10\", \"description^2.5\"]</b>,\n",
    "   <b>\"query_parser\": \"edismax\"</b>}  \n",
    "\n",
    "  {\"query\": \"the cat in the hat\",\n",
    "   \"params\": {\"defType\": \"edismax\",\n",
    "              \"qf\": \"title^10 description^2.5\"}}\n",
    "  </pre>\n",
    "\n",
    "Boosting on full phrase matching, on two-word phrases, and on three-word phrases is also a native feature of the edismax query parser:\n",
    "\n",
    "- Boost docs containing the exact phrase \"the cat in the hat\" in the `title` field:\n",
    "\n",
    "  <pre>\n",
    "  {\"query\": \"the cat in the hat\",\n",
    "   \"params\": {\"defType\": \"edismax\",\n",
    "              \"qf\": \"title description\",\n",
    "              <b>\"pf\": \"title\"</b>}}\n",
    "  </pre>\n",
    "\n",
    "- Boost docs containing \"the cat\", \"cat in\", \"in the\", or \"the hat\" in the `title` or `description` field:\n",
    "\n",
    "  <pre>\n",
    "  {\"query\": \"the cat in the hat\",\n",
    "   \"params\": {\"defType\": \"edismax\",\n",
    "              \"qf\": \"title description\",\n",
    "              <b>\"pf2\": \"title description\"</b>}}\n",
    "  </pre>\n",
    "\n",
    "- Boost docs containing \"the cat in\" or \"in the hat\" in the `description` field:\n",
    "\n",
    "  <pre>\n",
    "  {\"query\": \"the cat in the hat\",\n",
    "   \"params\": {\"defType\": \"edismax\",\n",
    "              \"qf\": \"title description\",\n",
    "              <b>\"pf3\": \"description\"</b>}}\n",
    "  </pre>\n",
    "\n",
    "\n",
    "Many of the relevancy boosting techniques will require constructing your own features leveraging function queries, however. For example, if we wanted to create a query that did nothing more than boost the relevance ranking of documents geographically closest to the user running the search (relevance based on distance away), we could issue the following query:\n",
    "\n",
    "  <pre>\n",
    "  {\"query\": \"*\",\n",
    "   \"sort\": \"<b>geodist(location, $user_latitude, $user_longitude) asc</b>\",\n",
    "   \"params\": {\"user_latitude\": 33.748,\n",
    "              \"user_longitude\": -84.39}}\n",
    "  </pre>\n",
    "\n",
    "The above is using the `sort` parameter to strictly order documents by the calculated value from the `geodist` function. This works great if we want to order results by a single feature, but what if we want to construct a more nuanced sort based upon multiple features? To accomplish this, we will update our query to include each of these function in each document's relevance calculation, and then sort by the relevance score:\n",
    "\n",
    "  <pre>\n",
    "  {\"query\": <b>\"{!func}scale(query($keywords),0,25) \n",
    "                {!func}recip(geodist($lat_long_field,$user_latitude,$user_longitude),1,25,1)\n",
    "                {!func}recip(ms(NOW/HOUR,modify_date),3.16e-11,25,1)\n",
    "                {!func}scale(popularity,0,25)\"</b>,\n",
    "   \"params\": {\"keywords\": \"basketball\",\n",
    "              \"lat_long_field\": \"location\",\n",
    "              \"user_latitude\": 33.748,\n",
    "              \"user_longitude\" -84.391}}\n",
    "  </pre>\n",
    "\n",
    "The above query does a few interesting things:\n",
    "- It constructs a query vector containing four features: BM25 Keywords relevance score (higher is better), geo distance (lower is better), publication date (newer is better), and popularity (higher is better).\n",
    "- Each of the feature values is scaled between 0 and 25 so that they are all comparable, with the best keyword/geo/publication date/popularity score getting a score of 25, and the worst getting a score close to zero.\n",
    "- Thus a \"perfect score\" would add up to 100 (25 + 25 + 25 + 25), and the worst score would be approximately 0\n",
    "- Since the relative contribution of 25 is specified as part of the query for each feature, we can easily change the weights of any feature on the fly to give preference to certain features in the final relevance calculation.\n",
    "\n",
    "With the above query, we have thus fully taken the relevance function into our own hand by modeling our relevance features and giving them weights. While this is very powerful, it still requires significant manual effort and testing to figure out which features matter for our domain, and what their relative weights should be. In chapter ? we will walk through building Machine-learned Ranking models to automatically make those decisions for us (a process known as \"Learning to Rank\"). For now, however, our goal was to ensure you understood the mechanics for modeling features in our query vectors, and controlling their weights. \n",
    "\n",
    "If you'd like to learn more about how to utilize function queries, I recommend reviewing chapter 7 of [Solr in Action](http://solrinaction.com), \"Complex Ranking Functions, for a much fuller exposition. For a full list of available function queries in Solr, you can also check out the [Solr Reference Guide](https://lucene.apache.org/solr/guide/8_3/function-queries.html).\n",
    "\n",
    "## Matching vs. Ranking\n",
    "Thus far, we've only really spoken of queries as feature vectors, and we've only discussed relevance ranking as a process of calculating and adding up scores for each each feature (keyword or function) in the query. This may seem a bit strange, since most search books start with coverage of matching keywords in the search engine's inverted index and filter result sets well before discussing relevance.\n",
    "\n",
    "We've delayed the discussion of \"filtering\" results until this point on purpose, however, in order to frame what search engines do as two explicitly different actions:\n",
    "1) Matching: Filtering results to a known set of possible answers\n",
    "2) Ranking: Ordering all of the possible answers by relevance\n",
    "\n",
    "In reality, we can completely skip step 1 (filtering) and we'd still see the exact same results on page one (and for many pages), since the most relevant results show up first. If you think back to chapter 2, we even saw some vector scoring calculations (comparing feature vectors for food items - i.e. \"apple juice\" vs. \"donut\") that were unable to filter results and had to try to score every document to determine which ones to return based upon relevance.\n",
    "\n",
    "So if the initial Matching phase is really optional, then why do it at all? The simple answer is that it is primarily a performance optimization. Instead of iterating through every single document and calculating a relevance score, by filtering the initial result set to a smaller set of documents which are logical matches, we can significantly speed up both our relevance calculations and the overall response time of our search engine.\n",
    "\n",
    "Of course, there are some additional benefits to filtering our results sets, in that the total document count is reduced and we can provide analytics (facets) on the set of logically-matching documents in order to help the user further explore and refind their results set.\n",
    "\n",
    "But when thinking about constructing relevance functions, like we did in the last section, the idea of filtering and scoring can often get mixed up, particularly since Solr itself mixes concerns in the query parameter.\n",
    "\n",
    "### Separating Concerns: Filtering vs. Scoring\n",
    "Solr has two primary ways to control filtering and scoring, the \"query\" (`q` parameter) and the \"filters\" (`fq` parameters). Consider the following request:\n",
    "<pre>\n",
    "{\"query\": \"the cat in the hat\",\n",
    " \"query_fields\": [\"description\"],\n",
    " \"filters\": [(\"category\", \"books\"), (\"audience\", \"kid\")],\n",
    " \"query_parser\": \"edismax\",\n",
    " \"min_match\": \"100%\"}\n",
    "\n",
    "{\"query\": \"the cat in the hat\",\n",
    " \"params\": {\"fq\": [\"category:books\", \"audience:kid\"],\n",
    "            \"qf\": [\"description\"],\n",
    "            \"mm\": \"100%\",\n",
    "            \"defType\": \"edismax\"}}\n",
    "</pre>\n",
    "\n",
    "In this query, Solr is being instructed to filter the possible result set down to only documents with a value of `books` in the `category` field and also a value of `kid` in the `audience` field. In addition to those filters, however, the query itself also acts as a filter, so the result set gets further filtered down to only documents also containing (100%) of the values `the`, `cat`, `in`, `the`, and `hat` in the `description` field.\n",
    "\n",
    "The logical difference between the `q` and `fq` parameters is that the `fq` only acts as a filter, whereas the `q` acts as BOTH a filter and feature vector for relevance ranking. This is useful default behavior for queries, but mixing the concerns of filtering and scoring in the same `q` parameter can be suboptimal, especially if we're simply trying to manipulate the relevance calculation and not arbitrarily removing results from our document set.\n",
    "\n",
    "There are a few ways to address this:\n",
    "1. Model the \"q\" parameter as a function (functions only count toward relevance and do not filter) \n",
    "  <pre>\n",
    "  {<b>\"q\": '{!func}query(\"{!type=edismax qf=description mm=100% v=$user_query}\")'</b>,\n",
    "   \"params\": {\"fq\": \"{!cache=false v=$user_query}\",\n",
    "              \"user_query\": \"the cat in the hat\"}}\n",
    "  </pre>\n",
    "\n",
    "2. Make your query match all documents (no filtering or scoring) and apply a Boost Query (`bq`) parameter to incluence relevance without scoring \n",
    "  <pre>\n",
    "  {<b>\"q\": \"*:*\"</b>,\n",
    "   \"params\": {\"fq\": \"{!cache=false v=$user_query}\",\n",
    "              <b>\"bq\": \"{!type=edismax qf=description mm=100% v=$user_quer}\"</b>,\n",
    "              \"user_query\": \"the cat in the hat\"}}\n",
    "  </pre>\n",
    "\n",
    "Both of these approaches are logically equivalent, but we'll go with option 2 throughout this book since it is a bit cleaner to use the dedicated `bq` parameter which was designed to contribute toward the relevance calculation without filtering. [BIG TODO: Verify that multiplicative boosts still work when specifying a bq= and a boost= using edismax."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Multiplicative vs. Additive Boosting\n",
    "One last topic to address concerning how we control our relevance functions is the idea of using multiplicative vs. additive boosting of relevance features.\n",
    "\n",
    "In all of our examples to this point, we have added multiple features together into our query vector to contribute to the score. For example, the following queries will all yield equivalent relevance calculations assuming they are all filtered down to the same result set (i.e. `fq=the cat in the hat`):\n",
    "- <pre>{\"query\": \"the cat in the hat\"}</pre>\n",
    "- <pre>{\"query\": '{!func}query(\"the\") {!func}query(\"cat\") {!func}query(\"in\") {!func}query(\"the\") {!func}query(\"hat\")'}</pre>\n",
    "- <pre>{\"query\": \"the cat\", \"params\": {\"bq\": \"in the hat\"}}</pre>\n",
    "- <pre>{\"query\": \"*:*\", \"params\": {\"bq\": \"the cat in the hat\"}}</pre>\n",
    "- <pre>{\"query\": '{!func}query(\"the cat in the hat\")'}</pre>\n",
    "\n",
    "This kind of relevance is known as Additive Boosting, and maps well to our concept of a query as nothing more than a vector of features in the query that need to have their similarity compared across documents.\n",
    "\n",
    "In many cases, however, we are likely to want to specify Multiplicative boosts as part of our relevance calculations. Instead of adding additional features to our vector, multiplicative boosts increase the relevance of an entire document by multiplying the document's score by some multiplier.\n",
    "\n",
    "For example, if we wanted to query for `the cat in the hat`, but wanted the popularity of documents (those with a higher number in the `popularity` field) to have a less constrained effect, we can't easily do this by just adding another feature into our query vector - at least not without modifying the weights of all the other features, plus any additional normalization that may be applied by the BM25 ranking function. If we wanted to apply multiple boosts like this (for example, boosting both on popularity AND on publication date), then the option of modeling this as an additive boost becomes unreasonably complex and harder to control.\n",
    "\n",
    "In Figure X, we were able to successfully utilize additive boosting by explicitly constraining the minumum and maximum values each feature in our query vector so that each feature provided a known contribution to the overall relevance function.\n",
    "\n",
    "Multiplicative boosting enables boosts to \"pile up\" upon each other, however, because each of the boosts is multiplied agains the overall relevance score for the document, resulting in a much fuzzier match and preventing the need for the kind of tight constraints we had to supply for our additive boost example.\n",
    "\n",
    "To supply a multiplicative boost, you can either use the `{!boost}` query parser in your query vector or, if you are using the `edismax` query parser, the simplified `boost` query param. For example, to multiple a document's relevance score by ten times the value in the popularity field, you would do either: \n",
    "  <pre>\n",
    "  {\"query\": \"the cat in the hat\",\n",
    "   \"params\": {\"defType\": \"edismax\",\n",
    "              <b>\"boost\": \"mul(popularity,10)\"</b>}}</pre>\n",
    "OR\n",
    "  <pre>\n",
    "  {\"query\": \"<b>{!boost b=mul(popularity,10)}</b>the cat in the hat\"}\n",
    "  </pre>\n",
    "\n",
    "In general, multiplicative boosts enable you greater flexibility to combine different relevance features without having to explicitly pre-define and exact relevance formula accounting for every potential contributing factor. On the other hand, this flexibility can lead to unexpected consequences if the multiplicative boosts values for particular features get too high and overshadow other features. Additive boosts can be a pain to manage, because you have to explicitly scale them so that they can be combined together and maintain a predictible contribution to the overall score, but once you've done this you maintain tight control over the relevance scoring calculation and range of scores.\n",
    "\n",
    "Both additive and multiplicative boosts can be useful in different scenarios, so it's best to consider the problem at hand and experiment with what gets you the best results."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Up next: Chapter 4 - [Crowdsourced Relevance](../ch04/1.setting-up-the-retrotech-dataset.ipynb)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
